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PREFACE 



The Research on Evaluation Program is a Northwest Regional 
Educational Laboratory project of research, development, testing, 
and training designed to create new evaluation methodologies for 
use in education. This document is one of a series of papers and 
reports produced by program staff, visiting scholars, adjunct 
scholars, and project collaborators — all members of a cooperative 
network of colleagues working on the development of new 
methodologies* 

How are SEA and lea evaluation-research-assessment units using 
computers in their work? What hardware and software arrangements 
are proving most effective? These and related questions are 
addressed in this report of a national phone survey of SEA and 
LEA unit directors. 

Nick L. Smith, Editor 
Paper and Report Series 
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APPLICATIONS OP COI-IPUTERS IN LEA AND SEA EVALUATION UNITS 



Study Overview and Purpose 

h 
k 

This paper addresses tho issues involved i^- the current 
application of computers, both microcomputers and mainframe, by 
LEA and SEA evaluation-research-assessment departments. This 
paper presents the results of a recent survey as well as 
integrates some of the salient points from existing publications 
on the use of computers by evaluators. This paper focuses on 
(1) the range of applications of computers, (2) problems in 
application, and (3) primary needs in the utilization of 
computers. 

Presently no summary is available concerning the use of 
computers by the evaluation-research-assessment units of local 
education agencies ^LEAs) and state education agencies (SEAs) • 
Information pertaining. to LEA/ SEA evaluation units' use of 
computers is based primai.!-^ on personal experience and anecdotal 
reports* Survey studieii have been conducted focusing on the size 
and function of state level evaluation units (Smith, 1S84) and 
even national de .criptive studies of evaluation practice (Boruch 
and Cordray, 1980; Raizen and Rossi, 1981). From these 
investigations, it appears that SEA evaluation units have 
undergone a dramatic reduction in personnel, with fewer 
evaluations being conducted. 

The median PTE in 1978 was 7, but only 3.5 in 

1983 • . • A median of 24.5 evaluations were conducted 

in 1978, but only. 8. 5 in 1983. 

(Smith, 1984, p. 3) 



Given this reduction in staff, what impact has the arrival of 
computers had? what are the profiles of computer use? Where is 
their application particularly successful? The context and 
status of current practice will provide a useful basis for 
addressing these questions and projecting future trends. 

The first section of this paper describes the population 
surveyed* The second part outlines the study design, and the 
final section sumiparizes the results of this research. 



Study Design 

The intent the investigation reported here was to study 
the current application of computers tc the task of evaluation- 
research by SEA and LEA staff. The study had three major 
purposes: (1) to document current practices, (2) to identify 
problem areas, and (3) to investigate future trends. 

Forty interviews were conducted for this research: 20 inter- 
views of SEA research-evaluation units, and 20 interviews of LEA 
research-evaluation units. The following criteria were used in 
the selection process. 

The Target Population 

A definition of SEA research and evaluation units is 
difficult to ascertain. There is considerable variability in 
evaluation practices at both LEA and SEA levels (Caulley and 
Smith, 1980), For example, some state-level evaluation and 
research departments engage in a wide range of activities, such 
as consultation/ technical assistance, evaluation studies, needs 
assessment, information provision, program monitoring, planning, 
policy formation, etc. On the other hand, other state 
departments restrict their activities to state-level testing or 
other singular requirements. 



o 8 
ERIC 



One state, for examplef has an office of research, 
evaluation, and testing, which is responsible for the 
evaluation of Title IV-C, while responsibility for the 
evaluation of Title I, Special Education, and Vocational 
Education is housed elsewhere in the agency. 

(Bracey, 1982, p. 13) 

State compliance with federal laws, regulations, and court 
decisions is the responsibility of state departments of 
education. The influence from the federal level is extensive, 
even to the point of specifying which evaluation procedures are 
used and which programs are ultimately evaluated (Smith, 1982). 

In addition to the federal influence, there are various 
elements within state governments which influence SEA 
evaluation. It is at this level that the special interests of 
legislators and the role of public opinion define the functions 
and activities of research-evaluation units. Foremost in 
directing state level agencies are statewide testing laws and 
state minirauin competency graduation requirements. 

Obviously these legislature-designed evaluation tasks can 
create considerable difficulties for SEA evaluation 
personnel who must attempt to perform such evaluations. 
These legislative mandates may change the nature of 
evaluation for a given program or have even more wide 
ranging impact. • . Legislative mandated evaluations 
frequently involve short timelines . . . (and] the need 
to use evaluation data to justify continuation of desired 
educational programs influence the nature and timing of 
evaluation studies. 

(Smith, 1982, p. 11) 

The decision rule used in this research to define what 
constituted an SEA research-evaluation unit was based on the type 
of unit activity and the centralization of the activity* 
Agencies chosen participated in three types of activities: 
evaluation, research, and assessment*. These activities were 
operationally defined to include monitoring, assessment, 
planning, programmatic evaluation, research synthesis and testing 
management. Further, the units considered for study were 
centralized and operated at one location within the state 
system. This classification was therefore based on both function 



and organizational structure. Those states which did not have 
centralized SEA evaluation-research units were excluded from this 
study • 

The decision rule for LEA' unit inclusion was similar, LEAs 
having over 10^000 students and a centralized evaluation- 
assessment unit were included in the target population. Their 
activitieb dre characterized by assessment, monitoring, testing 
management, technical assistance on test data, and policy 
development. The role of assessment practices was more apparent 
for LEA units. Therefore, SEA units will be described as 
evaluation-research units, while the LEA units are referred to as 
evaluation-assessment units, 

SEA Sample Selection 

From a previously constructed master list of state-level 
evaluation departments^, it was determined that 37 had centralized 
units. This list (dated 1978) contained the most current 
information available on centralized SEA evaluation departments. 
The total number of SEA units selected for this survey was 20, 
In the initial sample, the 6 states within the Northwest Regional 
Educational Laboratory region were included and a random 
selection of 14 of the remaining 31 states was made. Some states 
were found to have been misclassif ied as having centralized 
units, when they actually did not. These states were removed 
from the sample and replacements were selected. One case. New 
Mexico, was misclassif ied as not having a unit when it did. 

Corrections to the master list were made as follows: Georgia 
and Maryland were misclassif ied as having state^level centralized 
evaluation-research units, but presently they do not; New Mexico 
does have a state level unit. 

The map on the following page shows those states that were 
randomly selected. There appears to be a representative 
geographic distribution of states. 
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STATES PARTICIPATING IN NATIONAL STUDY 
ON COMPUTER APPPLICATIONS BY EVALUATION UNITS 



MAINE 



ALASKA 

• A 




SEA selected 

Alaska 

Washington 

Oregon 

California 

Nevada 

Utah 

Montana 

Wyoming 

Colorado 

New Mexico 



North Dakota 

Texas 

Illinois 

Mississippi 

Virginia 

Florida 

Pennsylvania 

Hawaii 

Michigan 

South Carolina 



LEA selectef^ = A 

Anchorage 

Annapolis 

Boston 

Buffalo 

Cleveland 

Fresno 

Washington* D. C. 
Pensacola 
Houston 
Long Beach 



Memphis 

Minneapolis 

Norfolk 

Omaha 

New Orleans 

Philadelphia 

Portland 

Seattle 

Tucson 

Tulsa 
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All states in the Northwest Region were included in the study 
because the Laboratory has a particular responsibility to this 
population. The six regional states are Oregon, Washington, 
Idaho, Montana, Alaska, and Hawaii, Idaho did not have a state- 
level centralized evaluation-research unit. Therefore, the final 
regional sample was 5 and a replacement was selected to insure a 
total of 20 states fitting the criteria. From a population of 36 
qualifying states, the total sample selected for the study was 5 
regional states plus 15 randomly selected other states, 

LEA Sample Selection 

The LEA sample selection process was less complicated than 
the SEA sample selection because the national master list of LEAs 
was more current than the SEA roaster list. This list was a 
random selection of 75 LEAs drawn froro a Center for the Study of 
Evaluation (UCLA) list of 510 LEAs with a 10,000 or more student 
population, A random selection of 20 froro the 75 vas made* 

The selection was similar to the one used in SEA selection^. 
If an LEA unit did not have a centralized evaluation-research 
component, then it was excluded and a random replacement was made* 

From the original 20 selectecS, 2 regional LEAs did not have 
evaluation-research units, and replacements were selected. The 
total LEAs with evaluation-assessment units sampled was 20, 3 
within the Northwest region and 17 throughout the nation* The 
map indicates the cities selected for LEA representation. 

Survey Design 

A phone survey was conducted of the directors or assistant 
directors of evaluation units in 20 state departments of 
education and 20 local school districts. Respondents were 
interviewed in February and March of 1985, 
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A semi- structured interview instrument was developed and 
revised on the basis of pilot trials and expert review (see copy 
of survey instrument in the appendix) • The interview required 
between 20-30 minutes, and there was 100 percent completion of 
all 40 interviews. 

The benefit of telephone interviewing was that it could be 
easily adapted to the respondent's schedule and cculd be 
completed within the study timeframe (1 month). It also allowed 
for informal inquiry about technical questions. 

At the time of the initial interviews, the focus of the study 
was on the use of microcomputers. It became immediately apparent 
that focusing only on microcomputers imposed an unrealistic 
limitation on the study of unit practice. If the study were to 
fully understand the role of microcomputers, it had to also 
address the use of mainframe and terminal computer systems. 
Microcomputer use must be examined within the full context of 
computer use to determine advantages, disadvantages, and types of 
application. 

In order to develop a profile of computer use, respondents 
were asked, "Does your unit use computers in their work? If so, 
please describe your system." 

Hardware arrangements were recorded as well as often-used 
software packages. Mainframe functions and packages were also 
inventoried. 



SEA Results 

The section on results is divided into two parts: SEA 
profiles by region and nation and LEA profiles by region and 
nation. Responses to questions 1-5 on the survey instrument are 
summarized. These questions refer to the profile or form of 
computer application. The second grouping is made from the 
responses to questions 6-8 which ask about problems and proposed 
solutions. A third cluster relates to the answers of a more 
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general inquiry on advantages of computers, their future, and 
other concerns pertaining to the use of computers by 
evalua.tion*-research units* 

SEA Profile — Regional 

The computing functions available to the SEA evaluation- 
research unit were responsive to the unit's main tasks. The main 
tacks of most SEA units incl'ided the provision of evaluation and 
technical assistance, monitoring projects, and organizing needs 
assessnient. computing functions to accc^mplish th^se tasks 
inclucied data processing, statistics, word processing, and 
spreadsheets, computer use profiles refer to the organization of 
computer use by hardware. The four basic models are: 

Model 1 - only microcomputers 

Model 2 - mainframe with »i:erminals 
no microcomputer 

Model 3 - both microcomputers and mainframes {two separate 
systems) 

Model 4 - microcomputers interfacing with mainframes 

All of the five Northwest region research and evaluation 
units had computer services. Computer use Models 1, 2, and 4 
were represented. Two states with Model 2 had budgeted for 
interf acting microcomputers for this year. The Model 4, in which 
the microcomputers can interface with the mainframe, appears to 
be the desired model for the Northwest region as well as for most 
other states. 

Most SEA units required the flexibility and access of 
microcoirputers and the storage and power of a mainframe? together 
these two computer resources can function in a hignly compatible 
mode. The advantage of such an interactive system is that an 
evaluator can maximize each of the technological resources (the 
mainframe and the microcomputer). Mainframes are particularly 
useful for some tasks, while microcomputers are oetter suited for 
others « 
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The tasks generally addressed by microcomputers are data base 
management, spreadsheet development, and graphics • Mainframes 
are used for statistics and data base* in all the state 
education agency (SEA) units surveyed, the word processing tasks 
were performed on "dedicated stations'* or terminals, or the 
mainframe by clerical personnel. 

Two regional states represented by Model 1 disdained the 
application of a mainframe, while the other three SEA units noted 
that the mainframe was used for specific tasks, such as 
statistics and data base management* The major advantage of the 
mainframe was its storage and power, but, overall, the 
disadvantages of wait-time, keyboarding, and lack of control 
outweighed the advantages. The n.ost responsive system was 
considered to be Model 4. 

The microcomputer software used by the regional SEA 
evaluation-research units included Visicalc, Lotus 1-2-3, 
d base II, Wordstar, and SuperCal 2. The types of hardware 
represented were IBM PC, IBM XT, Apple lie, TRS 100. 

The salient advantages of using computers — both 
microcomputers and mainframe — were identified as speed, 
flexibility, and access. These three characteristics were 
consistently rated as the most important feature of computers. 
The second order of importance, which became the first order once 
computers are taken for granted, was the type of questions that 
one can ask. The following statement was repeated by almost all 
persons interviewed: 

Computers allow one to ask "what if" questions. Before, 
we did not have the manpower to attempt asking a "what 
if" question. Now, we have much more opportunity to do 
intensive exploration of the data. We can try out 
questions, and hypothesize, look at the data, and have 
more control over the process. 

The answers to the last question, which concerned future 
needs, revealed the critical importance of linkages. All SEA 
unit respondents articulated their desire to have a system in 
which local districts could have access to state-level 
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information, and vice-versa. This shared information base would 
be a major advancement and was considered as a way to markedly 
increase productivity. 

In summary, computer application represented by Model 4 
represents the "state of the art" of computer use profile. In 
other words, this form is a blended system: both microcomputer 
and mainframe; in addition, the microcomputer interfaces with the 
ipainframe. In computer terminology, this allows for a "smart" 
microcomputer. Computer application as described by Model 4 is 
complementary and comprehensive, and is responsive to the needs 
of regional SEA evaluation-research units. 

SEA Profile — National 

It is interesting that the four models identified within 
NWREL regional states appear to cover the range of applications 
throughout the nation. In addition to these models, three SEA 
evaluation-research units had minicomputers, although in two of 
the three states, the unit directors said that these 
"intermediary" computers would soon be phased out. 

Table 1 presents the profile of the twenty SEA 
evaluation-research-assessment units sampled. Information is 
provided on type of models and number of professionals within the 
unit. 



Table 1 

Survey of SEA Evaluation-Research Units 
N=20 



Model 4 

Model 1 Model 2 Model 3 (Interfacing 

(aiccos only) (mainfrane) (micros & main) micros & main) 



No. of states 



Range of pro- 
fessional staff 1-3 4-21 3-35 5-55 

Average number 

of professionals 2.3 5.2 15.1 20.3 
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Of the seven state units representative of Model 3, three of 
them placed "high priority" on developing interfacing 
capability. They indicated that this would be accomplished 
within the year. The five with Model 4 generally (1) expressed 
relief that the interface connection process was completed, 
(2) are now looking for appropriate software to upload and 
download specific data, (3) found chat learning how to operate 
the interface process required at a minimum 6 months and in some 
cases almost a year, and (4) are now quite pleased with the 
results. The following statement by a director of evaluation in 
a western state department of education is representative of the 
remarks made by other directors. 

It was a great headache to get oar IBM mainframe (4341) 
to interact with our micros. (Apple 11+) . We spent over 
1 year in learning time — I mean mastering the upload and 
download without losing the data. We have ordered two 
more microcomputers: IBM PC and IBM XT and we hope that 
the updated terminal simulation package will provide us 
with a more compatible and workable system. Working out 
the special job control language from the Apple terminal 
was a real challenge. 

Another respondent explained: 

It was certainly a learning problem — figuring out how to 
upload and download without losing data. But it has been 
invaluable for our purposes, because we now have access 
to data that has been certified as correct and is 
specific to my application — especially demographic 
information. 

However, the survey findings revealed examples of less 
exasperating situations. An evaluation-research unit director 
from a north central SEA notes: 

The conversion on our mainframe was accomplished at the 
same time we had the microcomputers installed. So the 
system was integrated from the beginning^ We are still 
working with some of the features, and it seems to be 
coming along okay. 

It is apparent from Table 1 that the SEA evaluation-research 
units with more personnel have more complex facilities. This 
phenomenon is characteristic of growth patterns of larger 
systems. However, many lesser populated units also intend to 
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adopt the computer use^ arrangements represented by Model 4. 
Again, the argument is based on the specific features of both 
systems and the advantages of interfacing access. 

All of the sampled SEA units reported that they had 
"dedicated" terminals specifically for word processing. The 
hardware equipment most represented was IBM 8100, Zerox 860, 
WANG, CPT, and AB Dick. Almost none of tne microcomputers were 
used as word processing stations. Some of the professional staff 
developed rough drafts on a microcomputer, but the overwhelming 
norm was to give the handwritten draft to the word processing 
staff. 

The hardware for the mainframes was almost uniformly supplied 
by IBM. The most represented model was the series 4300 (4331, 
4361, 4381) which are classified as intermediate in terms of 
function and memory. A few SEA units had IBM 3080 and one had 
3090, both of which are rated as large in terms of function, 
memory, and price. 

The major problems fell into two categories. THe first area 
of concern was staff training. Many directors of evaluation- 
research units reported their uncertainty about how to proceed 
with staff training. Some said they would not "push" for 
training, and their staff could make their own decisions about 
obtaining computer skills. On the other hand, some unit 
directors felt that a regular training ptogram should be 
instituted. Basically, there was no consensus on this topic. 

A more nomothetic issue was the problem of non-integrated 
statistical processes. For the majority of SEA units, most of 
the statistical methods they needed were resident in the 
mainframe, and almost invariably, the summary numbers had to be 
hand-entered when a microcomputer or word processor was used to 
prepare the final document. The statistical functions were 
uniformly performed Dy either SPSS (Statistical Package for 
Social Science) or SAS (Statistical Analysis System) . 

The problem of hand-entering was not as evident when the 
statistics were developed by programs on the microcomputers, 
because some microcomputer packages integrate statistical 
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sunuaaries with other functions, such as word processing. The most 
frequently used such programs were Lotus 1-2-3, and the new SPSS-X 
package for microcomputers. Visicalc was also another conunon 
choice. A few units had written programs in which the statistical 
summaries were integrated into the ot'ier functions of the program. 

It is interesting that maintenance was not a conspicuous 
problem area. As noted, most SEA units purchased IBM equipment and 
had good experience with their IBM service contract. Using one 
vendor seemed to be one method of handling maintenance. For 
example, one state used only WANGs and was very satisfied with them. 

Finally, the consummate advantage of microcomputers was the 
control that the evaluation-research unit directors and staff felt 
they had. Repeatedly, the respondents noted that by having a 
microcomputer one was not "wedded to someone to key in data" or 
"waited for the host computer because of overload by too many users 
or it was filled to capacity," One key comment exemplifies the 
distinctive merit of microcomputers: "We are no longer just 
reacting to the data we receive from the mainframe. We can go out 
and initiate our own study, and this changes the nature of 
evaluation." 

Suggestions of improvements to the system included requests for 
optical scan capability, integrated graphics, data tapes to replaca 
keyboarding, and district linkages. 

In summary, most computer systems used by evaluation-research 
units at state education agencies can be categorized by the four 
profiles. Of the 20 states sampled, 12 had both microcomputers and 
mainframes — Models 3 and 4. The "state of the art" appears to the 
system in which microcomputers can interface with the mainframe. 
The timely use of information is critical to evaluation-research 
units, and they have found the computer to be an invaluable asset. 
The morale, eroded in the past by reductions in personnel and 
budgets, seems to be improving. Many of the directors reported 
increased confidence, interest, and professionalism coincidental 
with the arrival of computers, particularly microcomputers. The 
incidence of "cyberphobia" was quite low and most everyone in the 
units seemed to have benefited from this responsive, accurate, and 
versatile technology. 
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LEA Results 

The major tasks performed by most local evaluation units are 
associated with district-wide testing programs (Gray, Caulley, 
and Smith, 1982). Typically, this process includes selection of 
tests, data analysis, and reporting results to school boards. 
Some local education agency (LEA) evaluation units have the 
opportunity to do surveys and needs assessment. But few 
evaluation unit directors reported innovative activities (Gray, 
Caulley, and Smith, 1982). Generally, even the most active LEA 
evaluation units are circumscribed by financial constraints. 

Several LEAs noted that there is a shifting from 
evaluation to monitoring as funds dry up and monies are 
allocated to tangible activities, e.g., equipment and 
materials. A loosening up of evaluation requirements 
from SEA has also been reflected at the LEA level. 

(Gray, Caulley, and Smith, 1982, p, 35) 

Computers are essential for operating any testing program. 
LEA evaluation units also use computers for proposal writing? 
planning, data management, and analysis, as well as reporting. 

Twenty LEA evaluation units were selected. The survey 
corapletion rate was 100 percent. Again, the results are 
presented by regional LEAs, and Dy national LEAs. 

At the LEA level, the evaluation-assessment functions could 
be organized by units, but in the agencies interviewed tlose 
functions were often imbedded within other organizational* 
arrangements. Typically, there was no unit director, rither 
there was a program coordinator with a small sta£/< wose tasks 
included testing, evaluation, needs assessment, but almost no 
research. Generally, these staff members were the respondents to 
the LEA survey. 

LEA Prof ile- Regional 

Of the six regional LEA units sampled, three had 
evaluation-assessment services. One was represented by Model 3, 
one by Model 4, and one by Model 3 that intended to develop 
interface capability this year. The staff in all three LEA units 
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were very interested in the application of computers, Tv/o 
respondents were extremely knowledgeable and discussed the 
options of video disc technology, optical scanners, graphic 
plotters, and telecommunications. 

Most of the microcomputers were IBM types (PC, XT), while the 
mainframes were IBM 4300 series, VOX series, and Honeywell. 
Mainframes were generally used for data management and 
statistics; the micLOcomputers were used for specific analyses, 
spreadsheets, word processing, and telecommunications. 

The microcomputer software programs used most frequently by 
regional LEA units were Lotus 1-2-3, Lotus Symphony, and 
DB Master. Generally, the statistical calculations were 
performed on the mainframe using SPSS, and re-entered at the 
microcomputer level. 

The attention to the role of microcomputers is increasing as 
microcomputers become more powerful. The goal of every 
evaluation-research unit surveyed was to have an IBM PC XT (or 
AT) in each of the schools. The respondents believed that this 
level of microcomputer was powerful enough to address most all of 
for the needs of both the schools and LEA units. All of the 
microcomputers would be networked, then data could be input by 
microcomputer to the mainframe, calculations done and returned to 
the microcomputer, therefore not tying up the mainframe. 

One of the biggest concerns was the ability to access 
particular data. Item banks were being developed by districts in 
order to access directly an exact level of information. A 
comment by a program coordinator of an LEA unit illustrates this; 

We need to pull out specific information — for instance, 
how many kids who have been in a Chapter 1 program for 3 
years have improved test scores or improved attendance? 

The respondents were excited about the future of computer 
technology. Most of them considered Model 4 essential to 
accomplishing their tasks* One respondent commented 

with our budget reductions, the only way we can becjin to 
fulfill our purpose — especially reports to the scnool 
board — is by having computers. What I would really like 
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is for the decision makers to understand the necessity of 
computers, especially microcomputers • We couldn't 
operate without them. 



LEA Profile — National 

Seventeen LEAs were randomly selected • National 
representation is similar to the SEA selection • 

The function of most of the 17 LEA evaluation-assessment 
units was similar to the three regional units • One task that was 
included for many of these units, in addition to testing and 
monitoring, was maintaining small batches of information, for 
example, attendance by group, or quartile performance by groups. 



Table 2 

Survey of LEA Evaluation-Assessment Units 

N=20 



Model 4 

Hodol 1 Model 2 Model 3 (interfacing 

(micros only) (malnfrane) (micros & main) micros & main) 

No, o£ states 0 3 9 8 

Range o£ pro- 
fessional staff 0 1-4 1-32 1-15 

Average number 

of professionals 0 3 8«1 5*7 



Average no, 
students In 

LSA district 0 61,600 69,800 80,000 



As with the SEA evaluation units studied, the LEA evaluation 
units' range of computer application is represented by the four 
models. It is interesting that more LEAs than SEAs had the 
Model 4 system. One might have presumed that, at the state 
level, resources for computers would be more available than at 
the local level. But, consistent with other reports, much of the 
development of computer resources has a grass roots basis 
(Gustafson, 1985). Therefore, it is not unusual to find certain 
local educational agencies with well developed computer systems. 
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Of those local districts having the Model 3 system, four had 
inunediate plans to develop interfacing capabilities of their 
equipment. The pressing need for this improvement was described 
clearly and seemed to be felt more acutely at the local level 
than at the state level* Many respondents reported the pressing 
need to have all the schools in one district linked. A 
respondent, from a local district in the eastern U* S. emphasized 
this problem: 

#• 

We have to have connections with each scnool. We have an 
urban situation and students move around a lot. 
Presently their files are hand-delivered. A hand- 
delivered system — using the mail, etc. —takes too much 
time, and there are the risks of losing the files. Often 
a student will need to be tested before classroom 
placement. This represents duplicated effort if we have 
his records, but they just have not been delivered. The 
problem of records not catching up in a timely manner is 
real for us. 

The hardware profile at the regional level is similar to that 
at the national level. Combining the regional data with 
national, IBM mainframe was represented in 70 percent of the 
samples, the model usually being the 4300 series • There was 
greater variability in the choice of microcomputers. Although 
IBM PC, XT, and AT were used by 65 percent of the sample, Apple 
lie, TRS 80, Sperry, and Victor 9000 were also being used. 

Unlike the state level units, about one-third of the LEA 
evaluators used their microcomputers foe word processing. The 
word processing programs included Wordstar, Volkswritsr, 
Apple*- Writer, and Displaywriter . Other software programs 
selected most frequently were Lotus 1-2-3, d base III, and 
Visicalc 2. 

The range of microcomputer software is signifcantly greater 
in LEAs. The results from the SEA survey revealed a range of 
only 9 software programs, whereas LEAs reported a range of 14 
different, frequently used, programs. It is possible that word 
processing software may account for this greater variability; 
nonetheless, it is an interesting distinction between the two 
levels^ 
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Another differentiating factor is the U-shaped curve result 
relating to personnel and facilities. Findings from the LEA 
sample indicate that Model 3 units totaled more personnel than 
Model 4 units. In contrast, there is a linear trend (increasing) 
in the average numlDer of students per district from Model 1 to 
Model 4. 

The argument could be posited that technological tools, such 
as computers, are particularly useful for small staffs 'who are 
responsible for large numbers • Although ft was out of the scope 
of this research, these findings suggest that an evaluation of 
technological development in relation to responsibilities and 
size of district would be appropriate in the future. 

Some interesting variations in computer application appeared 
at the LEA level. Two situations are par ticularJ.y worthy of 
acknowledgement — one in which an LEA evaluation unit had a 
full-time programmer and one where data tapes were provided by 
prison inmates. In both these circumstances, unit directors 
noted these services were budgeted as line items. Unit directors 
had control over the process and could prioritize their needs, 
but delivery time was reported to be a problem. 

Indeed, the importance of control was addressed by all tne 
respondents, and related to this was the problem of "tying up the 
main." Due to frequent backups on the mainframes, microcomputers 
were considered more responsive for timely delivery. A comment 
from the evaluation unit coordinator of a large district in a 
southern community is illustrative: 

We sometimes have to do ad hoc reporting. We may get a 
request from the school board and often we have very 
little time to respond^ We can't submit a request to 
data processing, get a programmer, and develop a program, 
etc., and have the report ready. We need access to files 
all the time on an ad hoc basis. 

One respondent from a southwestern LEA evaluation unit 
explained: 

At one point I was running a lot of data on the mainframe 
and I was called on the phone by the superintendent. He 
said that data processing had to do tne payi;oll and asked 
if I could get off. Well, for the payroll, I certainly 
could. 




Curiously enough, ,few problems with software or hardware were 
numtioned by the LEA respondents. The problem of non-integrated 
statistics was noted, but since most LEA many reports were 
generated solely with a microcomputer and done entirely within 
one program, usually Lotus 1-2-3, integrating statistics into the 
text, was not an issue. 

Most of the statistics done on the mainframe were done with 
SPSS. The results, in most cases, would have to be re-entered at 
tne microcomputer level. Some LEAs had acquired the recently 
released microcomputer SPSS-X package and were in the initial 
phase of learning about its application* 

In terms of advantages, the response was uniformly — "we could 
not do our work without computers." One of the interesting, 
often noted, consequence of the accuracy and efficiency of 
computers (especially microcomputers) has been the upgrading of 
reports. A respondent from a midwestern local education agency 
commented: 

Staff and board decisions are based on better information. 
Their decisions are more profound in recent years because 
of having current data. Before, much of the decision 
making was gut reaction — seat of the pants — instinct. 
Now there is an educational process — they are much better 
informed. 

Under these circumstances, it is not surprising that many of 
the LEA evaluation-research units are excited about the 
opportunities provided by computer application. 

Yet one area of commonality with SEAs is the concern over 
staff development. LEA unit coordinators admitted that tney 
prefer an laissez-faire system of attaining computer skills. 
They encouraged their personnel to experiment with a variety of 
programs in order to find the ones that fit their needs best, out 
there was no uniform plan for training personnel. 

The final area of inquiry dealt with maintenance and future 
needs. It is interesting that the LEAs, with their more 
exploratory approach, did not have much of a problem with 
microcomputer maintenance as one mignt expect. Some dii.tricts 
had difficulty with manufacture support, but on the whole, the 
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reports indicated that microcomputers seem to be more reliaole 
than the mainframes. When an entire system like a mainframe was 
down, the impact was far greater than a single microcomputer 
going down. 

As far as software maintenance, there were virtually no 
comments about any problems. Certainly, there have been problems 
with losing data during the initial phase of interfacing, but 
once the process was working well, there seemed to be few 
problems with the software or the procedures. 

Finally, there was a remarkedly restrained description of 
future LEA computer needs. Most respondents were satisfied with 
their current system. They said they wanted more time to work 
with what they had before acquiring more. About the only 
improvement suggestions were for more IBM XT, IBM AT at schools 
in order to develop local school computer networks and 
telecommun ica t ions . 



Conclusion 

The intent of this researph was to conduct a current study of 
LEA and SEA application, problems, and needs of computer use by 
evaluation/research units. Some of the observations are as 
follows. 

Models 

It appears that computer application by these units at both 
state and local levels can be described by four models. The 
"state of the art" model is one in which microcomputers and 
mainframes are available and the microcomputers can interact with 
the mainframe. 
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Use and Effectiveness 

Generally, evaluators worKing in a variety of capacities 
assist with management decisions in a highly interactive 
process. Many of their reports go to legislators, school board 
members, and superintendents. The efficacy of computers is 
related to the evaluator's need for effective technical analysis 
and attendance to appropriate political considerations. This 
characterization is supported by other research (Smith, 1982) . 

That computers, especially microcomputers, emerged as key 
elements in such an environment is hardly surprising. The 
overriding objective of most evaluation-research units is to 
provide accurate, comprehensive information on testing, 
management and policy formation. However, in many instances 
there exists a cleft between 'old' and 'new' administration in 



the adoption of these technological aids. Many of the 
interviewees were concerned that their supervisors did not 
understand the consummate role of computers. Holding to the view 
that technological innovation is just a passing fad, many of the 
older administrators are in disagreement with their younger 
counterparts. Hence, there is a real need for explaining the 
value of computer technology to a wide audience, especially to 
the "old guard." 

With wider use of computers will come vast changes in 
comniuni cat ions. Most of the districts and states are preparing 
for this development. Five evaluation unit directors said they 
were awaiting an affordable voice synthesizer. Many now have 
electronic mail. Telecommunications is being explored by 
virtually every unit in the survey. In effect, each unit is 
orchestrating further application of computers, particularly in 
the area of communications. 

Yet, despite liberating qualities of technology, there are 
some major issues for organizational structure and human 
interaction brought about by increased computer use. As has been 
noted, staff training is a central concern. The survey results 
indicate no consensus on this. The alternatives range from 
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mandated training to ad hoc experience. Other research has 
indicated that there is an inherent reduction in hierarchical 
structure when everyone has access to a shared knowledge base 
(Naisbett, 1982), Consequently/ evaluation-research units may 
experience reduction in buireaucratic structure* Certainly, the 
desire for more control and more access was keenly articulated by 
all interviewed. 

By having microcomputers, most units will be able to 
investigate special issues, do the statistics, and develop their 
own reports. This self-sufficiency may change the nature of 
evaluation-research units. 

Finally, it appears that LEA units are more proactive and 
innovative than SEA units. Local evaluators seemed to have a 
better grasp of their role. They understood the larger picture 
within which they fit. Their orientation was primarily centered 
on evaluation of district-wide testing programs, and they had the 
resources to thoroughly study this area. An interesting comment 
from an £ast coast LEA staff member summarized this difference: 

We can ask questions that the state level people can't. 
For instance, I developed a spreadsheet and this 
information triggered other questions; I mean really new 
questions. This was an exciting moment for the staff. 1 
don't see the state people getting excited. We are 
lucky, we can focus; at the state there are so many 
factors influencing their work that they often have 
difficulty m focusing in depth on one topic. 

Computers will impact evaluation functions. This already 
occurs at a technological level, but indications are that 
organizational and social coui-exts will be modified as well. By 
examining the nexus between computer application and the context 
of LEA and SEA evaluation-research units, more relevant evidence 
can be collected. Evaluators within LEAs and SEAs are using 
computers and will continue to do so. to improve our 
understanding of computer application, to identify patterns, and 
to make mora insigntful comparisons, continual attention should 
be devoted to the role of computers by educational evaluation 
units. 
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APPENDIX 



Interview Form 



Telephone Interview Date_ 
Conducted by Merilyn Coe, ROEP 
Person contacted 



j>osition_ 



L£A,SEA,IND 



Size of evaliiation unit 



Topic: Computer Use in Evaluation/Assessment 

!• Does yourunit use computers: micro 



yes 



2* He f often used 



* of time 



type__ 



main w/ terminal yes 
main type 



3 . Does your unit have dedicated terminals 

purpose 
type 

4. Do the micros interface with the main 



terminal type 
yes 



no 



yes 



5. Has there (is there) been a problem with the interfacing 

6. Could the problem be identified as: 

insufficient softwzure availability 
software/hardwcire incompatability 
time available 

technical issues'* assistance, 
other 

?• How has your unit resolved these (this) problem? 
8. Tasks : Micro Use or Main Use - and program 



no 



yes 



not use 

W/P 


occasionally use 


frequently use 


Data Base^ 
Hanaqement 






Statistics 






Spreadsheet 






Graphics 






Telecom 






Instrument Gen 






Other 







no 



no 



no 



9, How have you handled the problem of non- integrated statistical packages? 

10. What are some of the salient features of using computers in your work? 

11. If you were to upgrade or change your computer system, what would you include? 

12. Has there been a problem with maintencince? 

13. other issues? 
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